Automatically reload certificate and key PEM files #4372

tsaarni · 2022-05-12T14:55:55Z

This change adds support to hot-reload TLS certificate and key without requiring server restart. It makes it possible to automatically reload PEM files, PKCS#12 and JKS keystores, when they are configured as file paths.

Closes #3780

Why support for hot-reloading should be implemented in Vert.x core:

Currently the core provides API for thr user to configure KeyStore and certificate & key file paths. The files are loaded into memory and path information is discarded. The SSL context is associated with this in-memory KeyStore. Even if the underlying files would change, the static in-memory KeyStore, built by Vert.x core, remains static. It seems unavoidable that the code paths dealing with the files, construction of KeyStores, associating KeyStores with KeyManagers and finally with SSLContexts is impacted.

Alternative to the above could be not to use the the API that Vert.x provides for certificate or keystore path configuration, but instead user can provide their own custom KeyManager. This implies that #3780 would not be implemented in Vert.x but by each user should implement it by themselves.

The approach taken by this PR:

The change implements new key stores by extending JSSE KeyStoreSpi:

PemFileKeyStoreSpi loads and reloads PEM files, contructs single in-memory JSSE keystore from them and delegates operations to it.
KeyStoreFileSpi loads and reloads keystore files (PKCS#12 & JKS) and delegates operations to JSSE keystore.

The files are reloaded when they are modified. Modification is observed by checking the file modification timestamp, but only when certificates and keys are used (when KeyStoreSPI methods are called) and the interval is capped by cacheTtl, currently fixed to 1 check per second at most.

Previously, handling of certificates was centered around crafting KeyStore(s) from values. The benefit was that handling of credentials configured "by-value" and "by-path" were all normalized into single "by-value" use case. To be able to reload, the path information must be kept around. The "reloading keystores" are constructed "by-path" directly in PemKeyCertOptions.java and KeyStoreOptionsBase.java.

Previously selection of server certificate according to TLS SNI servername was implemented by custom-built logic by splitting each certificate in individual KeyStore(s) and KeyManager(s) and picking one of those according to the SNI information from Netty, when handling new TLS connection. The NewSunX509 version of KeyManager (from JDK 8) has support for selecting from multiple certificates in keystore according to SNI and other criterias, such as key type. This change proposes to change to JSSE's built-in SNI support. The side effect of this is that SNI support cannot be disabled anymore setSni(false) server option will not be effective: correct server certificate (according to SNI) will be returned from keystore when it previously could return incorrect one.

For detailed information of the proposed implementation see this project and associated design document.

Signed-off-by: Tero Saarni tero.saarni@est.tech

Note: Change stacks on top of #4388 PR branch in order to pass the tests. JSSE SNI server cert selection support requires valid certificate dates.

TODOs

Add new test cases for hot-reloading
Remove unused code after moving from custom-built SNI support to JSSE SNI support

pmlopes · 2022-05-20T06:33:07Z

src/main/java/io/vertx/core/net/impl/ReloadingPemKeyManager.java

+  }
+
+  private void pollForFileChanges() {
+    vertx.setPeriodic(FILE_POLLING_PERIOD_MSEC, id -> {


Why not use the JDK file watcher instead?

FileSystems.getDefault().newWatchService()

Thanks for reviewing!

I had another try before this PR with a WatchService and theads to run them master...Nordix:certificate-rotation.

My problem there is that I cannot find a reliable way to delete the watch filehandles and threads. The X509KeyManager / X509ExtendedKeyManager interface have no close method and therefore they are not closed explicitly. I ran out of inotify filehandles when running test suite, since the key managers and the watches they hold were not garbage collected. I'd appreciate any ideas here!

Now currently, I've been experimenting with another approach that avoids periodic polling:
When key or certificate is requested from key manager, return cached versions unless time-to-live period (of e.g. 1 second or something else) has passed. If cache TTL has passed, check file modify timestamp. If the files have been modified then reload.

Maybe another (secondary) reason to not use WatchService is that they are sensitive on how the files are modified. In that previous link with my experiments, I added an example in comments how Kubernetes does the atomic swap of the files https://github.com/Nordix/vert.x/blob/db145f3ba4c6735a80fa005df1f30bdffd611b1a/src/main/java/io/vertx/core/net/impl/FileWatcher.java#L48-L85. One size does not fit all, so some configurability might need to be raised up all the way to all applications (through the frameworks that wrap Vert.x).

Some of the certificates in test resources had unintentionally expired. They could cause test errors if used in tests that expects valid certificates. * Reformatted ssl.txt file into a shell script ssl.sh which can be executed to regenerate certificates and keystores. * Updated all test certificates with 30 year expiration period (was 3 years). Signed-off-by: Tero Saarni <tero.saarni@est.tech>

* Implemented KeyStore that can reload PEM and PKCS#12 / JKS files wihtout server restart. * Use NewSunX509 KeyManager which selects server certificate based on SNI in TLS handshake among other selection criterias. Signed-off-by: Tero Saarni <tero.saarni@est.tech>

tsaarni · 2022-06-07T06:38:29Z

I've updated the PR content and the description. I would greatly appreciate feedback, and comments wether you agree on the approach and would be interested in incorporating hot-reload functionality into Vert.x.

Note: Change stacks on top of #4388 PR branch in order to pass the tests. JSSE SNI server cert selection support requires valid certificate dates.

vietj · 2022-06-07T12:49:27Z

I will have a look

tsaarni · 2022-07-13T16:52:37Z

I will have a look

Friendly reminder, asking if you would have time for the review :)

On slightly related note, in case it would be useful for the review: I've created a separate project and a design doc showing the ideas from this PR as a standalone project.

vietj · 2022-07-13T17:05:42Z

ok, I would like to see if we can have it as something more decoupled from vertx core

tsaarni · 2022-07-13T18:17:12Z

ok, I would like to see if we can have it as something more decoupled from vertx core

Thanks! Curious to hear if you'd have ideas about this!

Since Vert.x is quite intensively manipulating KeyStores, it is already entangled with the code paths that would need to support hot-reload. Therefore it is bit difficult for me to figure out how decoupling could be achieved, That is, if #3780 is to be implemented in the first place. There is the "escape hatch" where application would not use Vert.X certificate loading support at all, but instead they would create and pass their own KeyManager, initialized with hot-reloading KeyStore.
I did not explore this path so far, as I thought it would be rather "heavy" approach to require applications to make the change - especially since some use Vert.x through additional layers like Quarkus.

tsaarni · 2022-07-22T20:40:23Z

I've updated the PR description according to the discussion above.

vietj · 2022-12-17T10:47:51Z

See #4568

tsaarni force-pushed the hot-certificate-reload branch from 47394b3 to e9bf920 Compare May 13, 2022 09:54

tsaarni mentioned this pull request May 13, 2022

Hot Reload for TLS Keystore quarkusio/quarkus#15926

Closed

pmlopes reviewed May 20, 2022

View reviewed changes

tsaarni added 3 commits May 27, 2022 20:31

Merge branch 'master'

e26ff35

tsaarni force-pushed the hot-certificate-reload branch from e9bf920 to f74cc76 Compare June 7, 2022 05:51

tsaarni mentioned this pull request Jun 29, 2022

Update expired test certificates #4388

Closed

This was referenced Aug 9, 2022

Hot Reload SSL Configuration #4452

Closed

Reloading SSL Configuration #4453

Closed

vietj mentioned this pull request Aug 23, 2022

Execute SSLHelper.validate as blocking #4441

Closed

Hakky54 mentioned this pull request Aug 30, 2022

You may like securitybuilder Hakky54/sslcontext-kickstart#212

Closed

vietj closed this Dec 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatically reload certificate and key PEM files #4372

Automatically reload certificate and key PEM files #4372

tsaarni commented May 12, 2022 •

edited

Loading

pmlopes May 20, 2022

tsaarni May 20, 2022 •

edited

Loading

tsaarni May 20, 2022

tsaarni commented Jun 7, 2022

vietj commented Jun 7, 2022

tsaarni commented Jul 13, 2022

vietj commented Jul 13, 2022

tsaarni commented Jul 13, 2022

tsaarni commented Jul 22, 2022

vietj commented Dec 17, 2022

Automatically reload certificate and key PEM files #4372

Automatically reload certificate and key PEM files #4372

Conversation

tsaarni commented May 12, 2022 • edited Loading

pmlopes May 20, 2022

Choose a reason for hiding this comment

tsaarni May 20, 2022 • edited Loading

Choose a reason for hiding this comment

tsaarni May 20, 2022

Choose a reason for hiding this comment

tsaarni commented Jun 7, 2022

vietj commented Jun 7, 2022

tsaarni commented Jul 13, 2022

vietj commented Jul 13, 2022

tsaarni commented Jul 13, 2022

tsaarni commented Jul 22, 2022

vietj commented Dec 17, 2022

tsaarni commented May 12, 2022 •

edited

Loading

tsaarni May 20, 2022 •

edited

Loading